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(57) Abstract 

The apparatus (10) is a color wide dynamic range apparatus which includes a filter (12) interposed immediately in front of reoccuning 
color elements so that each pixel represents a given color element for the scene. At least two exposure levels are taken of the scene and 
the pixel outputs are decoded to generate the video luminance and chrominance signals. The images of the at least two exposure levels are 
combined to form a final image. 
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COLOR WIDE DYNAMIC RANGE CAMERA 
ITS TNG A CWAPBB COUPT.EP PEVICF - WTTH MOSAIC FILTER 



raOSS-REFFRENCE TO RELATP D APPLICATIONS 

This application is a continuation-in-part of U.S. 
patent application Serial No. 07/795,350 filed November 20, 
1991 entitled "Color Wide Dynamic Range Camera", which is, 
5 in turn, a continuation-in-part of U.S. patent application 
Serial No. 07/388,547, filed August 23, 1989, now U.S. 
Patent No. 5,114,442. Additionally, this application is 
related to U.S. Patent No. 4,858,014 and currently pending 
U.S. patent application Serial No. 07/805,512, filed 
10 December 11, 1991. The disclosures of all of the above- 
. identified U.S. patents and patent applications are 
incorporated herein by reference. 

HACKGROUyn OF THE INVENTION 

15 Field of the Invention 

This invention pertains to video imagery and more 
particularly to apparatuses and techniques for providing 
enhancement of video color images. In particular, the 
present invention uses a four-color mosaic filter with a 

20 single chip CCD in conjunction with color wide dynamic 
range algorithms. It is also applicable, however, to other 
types of mosaic filters known in the art. 
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Description of the Prior 

Various types of video enhancement apparatuses and 
techniques have been proposed. Prior implementations of 
color wide dynamic range cameras, such as those disclosed 
5 in the above-identified parent applications hereto, have 
used a plurality of CCD chips to generate the image data 
for subsequent processing. The use of multiple CCD chips, 
however, adds to the complexity and cost of the instrument! 
Moreover, current consumer video cameras, i.e., camcorders, 
10 almost universally use a single CCD chip. Therefore, a 
single CCD implementation is required to use dynamic range 
enhancement algorithms in a camcorder. Single CCD chip 
implementations are similarly preferred for endoscopic 
applications. 



15 



20 



OBJECTS AMD SUMMARY QF THE IKVENTIOW 

It is therefore an object of this invention to provide 
a color wide dynamic range camera implemented with a single 
CCD chip. 

It is also an object of this invention to provide a 
color wide dynamic range camera which is adapted for use 
with a camcorder. 

It is a further object of this invention to provide a 
color wide dynamic range camera which is adapted for use 
25 with a conventional endoscope. 

These and other objects of the invention will be more 
apparent from the discussion below. 

SUMMARY OF THE TfJVRNTTfYU 

30 There is thus provided in accordance with the 

preferred embodiment of the present invention, video 
imaging apparatus including means for providing a plurality 
of video color images of a scene at different exposure 
levels using a single CCD chip, each color image being 

35 separated into several (e.g., four in the preferred 
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embodiment) different components prior to sensing by the 
CCD chip by way of a multiple color mosaic filter in front 
of the CCD chip. The pixel outputs are then decoded — 
subjected to specific mathematical operations by the 
5 processing electronics following the CCD output — to 
generate the video luminance and chrominance signals. 

The present invention integrates the digital 
processing of the mosaic color CCD data with ADAPTIVE 
SENSITIVITY™ dynamic range enhancement. This integration 
10 provides for a substantial savings in total' system 
processing hardware chip count and cost. It also permits 
better control of the color and detail production of the 
camera's video output. The mosaic storage format also 
provides for a unigue video image compression technique. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 
Further objects and advantages of the invention will 
become apparent from the following description and claims, 
and from the accompanying drawings, wherein: 
20 Figure 1 is a general block diagram of the present 

invention. 

Figure 2 is a representative illustration of the data 
image elements, with the size of the data image elements 
exaggerated . 

25 Figure 3 is a general block diagram of the long and 

short processing of the present invention. 

Figure 4 is a block diagram of the color path of the 
present invention. 

Figure 5 is a block diagram of the intensity path of 
3 0 the present invention. 

Figure 6 is a block diagram of the look-up table 
processing of the present invention. 

Figure 7 is a block diagram of the joint operations of 
the present invention. 
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Figure 8 is a block diagram of the differential color, 
intensity result block of the present invention. 

Figure 9 is a block diagram of the color suppression 
factor block of the present invention. 

Figure 10 is a block diagram of the color conversion 
block of the present invention. 

Figure li is a block diagram of the mosaic generation 
block of the present invention. 



DETAILEP DESCRIPTION OF THF PREFFPPFn EMBonTMBOTc 

Referring now to the drawings in detail wherein like 
numerals indicate like elements throughout the several 
views, one sees that Figure 1 is a block diagram of the 
apparatus 10 of the present invention. 

Apparatus 10 includes a mosaic filter 12 which is 
bonded to the front of CCD 14 (preferably a single chip), 
generally as part of the CCD integrated circuit 
manufacturing process. 

The alternating mosaic filter elements are cyan, 
magenta, yellow and green (wherein c 'cyan' - g 'green' -1 
B ' blue ' ; M 'magenta' = R 'red' + B; and Ye 'yellow' = 
R+G). when the CCD 14 charge output is read out the 
Photoelectric charges from vertically adjacent sensor 
elements of CCD 14 are combined in the analog shift 
25 register (not shown). The on-chip addition gives rise to 
a, 0, y and S elements as described below and as described 
m the Sony CCD 1992 Data Book (Sony part number ICX038AK) , 
as well as earlier editions. 

As shown in Figure 2, the mosaic complementary 
3 0 additive color image comprises alternating first and second 
rows of image data elements 18 A - is D wherein first rows 
include alternating a and 7 data elements (ie A and 18 
respectively), and wherein the second rows includ'e 
alternating /? and 6 data elements (is c and is 
35 respectively) . D ' 
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The a image data elements 18 A are an equal mixture of 
cyan plus green (i.e., C + G - B 'blue' + 2G 'green'). The 
7 image data elements 18„ are an equal mixture of magenta 
plus yellow (i.e., M + Ye = 2R 'red' + B + G) . Similarly, 
5 the 0 image data elements 18 c are an equal mixture of cyan 
plus magenta (i.e., c + M=2B + G + R) and the 6 image 
data elements 18 D are an equal mixture of green plus yellow 
(i.e. , G + Ye = 2G + R) . 

Those skilled in the art will recognize that: 

10 Y (i.e., intensity) = SL ^ - 

from which the definition of intensity (Y) in the red, 
15 green, blue (RGB) system may be derived: 

Y = R + 1.5G + B 

Of course, those skilled in the art will realize that 
other color combinations are equally applicable. Each 
mosaic element of filter 12 covers the sum of two adjacent 

2 0 pixel sensors of CCD 14 so that each pixel output~,of CCD 14 

is representative of one of the above color combinations 
given for the various image data elements 18. Four 
different monochromatic images, each representative of one 
color combination chosen from the colors of image data 
25 elements 18 A - 18 D of a given scene, are therefore generated 
by CCD 14. 

As can be further seen from Figure 1, apparatus 10 
includes four major functions as summarized below: 

1. Long/ short exposure processing: 

3 0 The first stages of the algorithm are performed on the 

two exposures (long/short) separately. The processing 
of each exposure is divided into two paths: 

a. Color path processing - evaluates color component 
for each pixel. 

35 

b. Intensity (Y) path processing - handles intensity 
information for each pixel. This includes point 
("DC") intensity and edge information. 
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Point processing: 
Each of the long/short exposure length processing 
functions (typically implemented on separate chips) 
outputs its point intensity information, obtained from 
the Y path processing, to four look-up tables (LUTs) . 
These tables determine the point intensity result of 
the two exposures, the normalized color weighting or 
color selection function and the saturation color 
suppression factor. This information serves the joint 
operation processing stage. The four LUTs are 
programmable, thus enabling operation with different 
functions when necessary. m an alternative embodi- 
ment, these LUTs may be replaced by a programmable, 
piecewise linear (PWL) or other digital function 
generator . 
Joint operations processing: 
Joint operations processing joins results produced by 
the long and short processing blocks, and results 
obtained from the functions implemented in the table 
processing LUTs, and evaluates the final output of the 
algorithm. The processing is divided into: 

evi!Ltt« mPO /l entS ^ an 1 Y result calculation - 
evaluates the final result of the color 

components and the intensity of each piJel. 

2!i 0r ™i U i!? re6Si0n fa . Ctor ca l<=ulation - evaluates 
the color suppression factor for each pixel 
based on both edges and saturation information.' 

dirlL^^ 810 ," processi »g ~ converts mosaic 
differential color space to RGB color space and 
produces RGB and Y/Cr/Cb outputs for each pixeJ 
Generate mosaic processing. 
Generate Mosaic processing converts RGB color space 
back to mosaic color space for each pixel. The mosaic 
information generated enables economical hardware 
storage of processed images. This information can be 
retrieved and replayed through the algorithm - in 



b. 



c. 
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Replay Mode — to produce RGB or Y/Cr/Cb output of the 
stored result. 

Referring now to Figure 1, similar to U.S. Patent No. 
5,144,442 and parent U.S. patent application Serial No. 
5 07/795,350 (the disclosures of which, again, along with 
U.S. patent application Serial No. 07/805,512 and U.S. 
Patent No. 4,858,014 are incorporated herein by reference) , 
apparatus 10 includes long/short processing as implemented 
by mosaic long exposure field block 20 and mosaic short 
10 exposure field block 22 which obtain, respectively, a long 
and a short exposure from CCD 14 in order to allow subse- 
quent processing by long exposure processing block 24 and 
short exposure processing block 26. The terms "long 11 and 
"short" exposures are used here generally to denote two 

15 image inputs to apparatus 10. In general, "long" is used 
to mean an input with a higher exposure level, and "short", 
a lower exposure level. The higher exposure may be 
generated in several ways, including longer integration 
time, typically obtained by controlling the "electronic 

20 shutter" of the CCD chip; higher gain in the analog 
amplifiers preceding digitization; or a larger mechanical 
iris opening or other external gating means. 

These two image inputs are usually generated by a 
single CCD chip, but may also be generated simultaneously 

25 by two separate, boresighted CCD chips, as disclosed in the 
aforementioned earlier applications. For the more common 
case in which the two inputs are generated by a single CCD 
chip, they may be generated either sequentially (as in the 
case of the first method above — integration time control) 

3 0 or concurrently (by using two input channels with different 
gain levels) . When a sequential method is used, field 
memories are required at the input to apparatus 10 (in 
blocks 2 0 and 22) to synchronize the data coming from the 
two sequential fields or frames. These memories are not 

35 needed in concurrent modes, except for purposes of 
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"freezing" the image for electronic, digital storage, 
switching logic incorporated in blocks 20 and 22 controls 
the data flow into and out of these field memories, 
depending on which mode (sequential or concurrent) is used.' 
Of course, this implementation could be expanded to more 
than two exposure levels. Blocks 24 and 26 may typically 
be provided on separate processing chips or incorporated 
together in a single chip. The processing for each 
exposure is divided into two paths: 

1. Color path processing - handles color information 
for each pixel (see color path block 28 in Figure 
3 and, in more detail, in Figure 4); and 

2. Intensity (Y) path processing - handles intensity 
information for each pixel (see Y path block 30 
in Figure 3 and, in more detail, in Figure 5). 

Additionally, as shown in more detail in Figure 3 
long/short exposure processing blocks 24, 26 include mosaic 
white balance block 32. 

Mosaic white balance block 32 receives the following 
field of information from long/ short exposure field blocks 
20, 22: 



25 



ol y a y a y 

0 S 0 6 0 6 

a 7 o y a y 

& € 0 S 0 6 

<* y a y <x y 

0 6 0 6 0 S 



30 



That is, the information from CCD 14 with the mosaic 
data order intact is received. 

After processing, the mosaic white balance block 32 
outputs color-corrected data values: 
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0wb 
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<*wb 
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Twb 


a«t 


7wb 


flwb 


*wb 


^Wb 




£wb 


*wb 


<*wb 




"wb 


7wb 


<*wb 


7wt, 


£wb 


^wb 


0wb 


^Wb 


A* 


*wb 



10 Mosaic white balance block 32 contains mosaic color 

balance functions. These functions may typically be 
implemented as eight mosaic white balance LUTs (look-up 
tables) . That is, for each exposure there is a set of four 
LUTs, one for each mosaic data type: a, 0, y, and 6. 

15 Independent calculation of white balance correction factors 
is performed for each exposure. This enables white 
balancing scenes where the observable parts of the two 
exposures are at different color temperatures. The LUTs 
may contain multiplicative correction factors which are 

2 0 evaluated as follows: 

From the definitions of a, jS, y, 6 and Y it follows, 
that for a white image (where by definition R=C=B) , the 
following relations should hold: 



25 


<*wh 






ft** 




30 




7 


35 
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Based on these relations correction factors can be 
calculated by enforcing these relations on the average of 
a white image: 



- ill 



10 8 - 



c 



1 * * 



20 6 

■r * Y 



c 4 - -4- 



25 where Y denotes selective average over Y in the given 
white image and a, J, y and 7 are the respective average 
values of a, B, y and S. Saturated or cutoff pixels are 
excluded from this average, since by definition, 

30 y = ° + 7 - B + ~6 

2 - 2 

the equations for the correction factors are: 
c„ = f * ( 1 f-f 



40 



7 



# * I 1 +- 



50 



f ) 



The LUT values are calculated by simple multiplication of 
each mosaic data type by its respective correction factor: 
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a wb « a * C a 
0 wb = fi * C, 
7wb = 7 * C T 

5 5 W b - * * C a 

In an alternative embodiment, these LUTs are replaced 
by digital multipliers. Furthermore, the LUTs may also be 
loaded with correction functions other than simple linear 
multiplicative factors. Alternatively, the mosaic balance 

10 correction factors can be computed based on four average 
signals, namely, a, 0, 7 and 6, instead of merely two of 
them as above. This alternative yields improved uniformity 
of color balance under difficult conditions. Alterna- 
tively, the white balance function may be done on the RGB 

15 color components in the color conversion block 78 
(described below) . 

Referring now to Figure 4, color path block 28 is 
shown in more detail. As previously stated, the input to 
color path block 28 is the image data cr wb , /J wb , y wbf <S wb after 

20 processing by mosaic white balance block 32. 

The initial processing of color path block 28 is 
performed by color difference evaluation block 34 which 
receives data a wb , (3^, 7 wbf S wb from mosaic white balance 
block 32 and calculates color difference components dr, db 

25 for each pixel in the array: 

dr dr dr . . . . = (7-a) (7-a) (7-a) .... 

db db db . . . . = (tf-0) (*-£) (6-0) .... 

dr dr dr . . . . « (7-a) (7-a) (7-a) .... 

db db db . . . . = (*-/3) (*-£) (6-0) . . . . 

3 0 wherein: 

dr e 7 - a , the differences between successive 
readings in even lines. 
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db e s - 0 , the differences between successive 
readings in odd lines. 

The correct evaluation of dr and db requires 
horizontal interpolation as described in the following 
equations : 

For each pixel (i,j) in even lines i: 

dr (3odd)=7wb(Dodd) . — 

Ar .f+ x_ 7wb(jeven-l) + 7wb ( j cveo+ 1 ) 

3cvct/ 2 <*wb(jev«,) 

where j is the pixel index along the line (here and 
henceforth) . 

For each pixel (i,j) in odd lines i: 

dbM ) —s r-i > ^(j^-i) + £ wb ( j^+i ) 

aD l JoddJ — 6wb(3o4l) = _ 



dbM > = ^wh(jevc.-l) + *wb(j CT „+D 
V J even / 



Color difference components dr, db are thereafter 
25 received by low-pass color component block 3 6 which 
calculates a low-pass color component dr„ or db,, for each 
pixel : 



30 dr^dr,,, 



dr.pdr.pdr,,, . . . . . (y-a)^ . 



dbjpdb^db^, . 
dr^dr^dr^, . 
dbjpdb^db^ . 



Block 36 performs horizontal low-pass filtering on dr^ 
and db„p calculated in block 34. This reduces color 
artifacts caused by interpolation. in a preferred 
embodiment, the low-pass filter width is five pixels and 
its coefficients are 14, %, %. The equations follow: 
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For pixels (i,j) in even lines i: 

dr(j-2) + 2*dr(j-l) + 2*dr(j) + 2*dr(j+l) + dr(j+2) 
dr^j) — 

5 8 

For pixels (i,j) in odd lines i: 

db(j-2) + .2*db(j-l) + 2*db(j) + 2*db(j+l) + db(j+2) 
db ip (j)= 

10 8 

Delay buffer 38 receives the output from low-pass 
color component block 36 and directs db^(i CTai -1) , dr^i^ - 
1), db,p(i CTm +1) and dr*( 1*4+1) to vertical interpolation 
block 40 and drCi^) and dbCi^) to multiplexer 42. 
15 Vertical interpolation block 4 0 receives the low-pass 

color components as described above and generates 
interpolated low-pass color components dr'^, in the odd 
numbered lines and db' ¥ in the even numbered lines: 

dbV db^ db',,, dbV 

2 0 ox'* drV dr'* dr'* 

db'„ db'.p db',p db'.p 

drV dr',, dr',p dr^ 



The equations follow: 

For even lines i: 

25 db,p(i-l,j) + db*(i+l,j) 

db' lp (i,j)= j 

For odd lines i: 

dr,p(i-i,j) + dr,p(i+i,j) 
30 dr' lp (i,j)= 2 
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The interpolated low-pass color components dr'^ db' 
are multiplexed with the original low-pass components dr*, 
db,, to give the color path output values dr and dp for each 
pixel. This function is performed by multiplexer 42, which 
separates the output received from delay buffer block 38 
and vertical interpolator block 40 into a first path 
including db*' ( ie¥ai ) and db^i^) and a second path including 
dr * (i«v«) and dr'^i^) . 

Referring now to Figure 5, which discloses in more 
detail the intensity (Y) processing block 30 shown in 
Figure 3, one sees that the input to intensity (Y) 
processing block 30 from mosaic white balance block 32 
(Figure 3) is received by intensity evaluation block 44 
which outputs computed intensity Y for each pixel. 

Since only one of the four data types (a, 0, \, S ) is 
present at any given pixel, the intensity evaluation block 
44 calculation is performed as follows (based on the prior 
definition of Y) : 

For pixels (i,j) i n even lines i: 

20 

25 Y(j««J 



15 



30 



35 



For pixels (i,j) in odd lines i: 



7(j) 


+ 


a(i-l) 




2 






+ 


y(i-l) 




2 




lines 


i: 






4 


0(1-1) 




2 






+ 


*(j-D 


2 



Y(jeva.) 

The output from intensity evaluation block 44 is 
received by delay buffer 46, generate output intensity 
block 48 and limit block 50. 

Delay buffer 46 is a delay line of two horizontal 
lanes, required for the 3x3 and 1x3 matrix transformations 
in Y path block 30. Together with the color path delay 
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buffer 38 and with Y path delay buffer 54, it may be 
implemented in a preferred embodiment in mosaic data space, 
operating on the input a, 0 , 7, 6 data before the intensity 
(Y) evaluation block 44 and color difference evaluation 
block 34. It is shown here schematically for clarity. 

vertical low-pass filter 52 receives intensity (Y) 
signals from the intensity evaluation block 44 as delayed 
by delay buffer 46. Block 52 generates the vertical low- 
pass intensity Y vlp defined as: 

Y VIp (i,j) = Y(i-l,j) + 2 Yg,i) + Y(i+l,j) 



The unfiltered intensity (Y) input will sometimes 
exhibit horizontal stripes, one pixel high in each field , 

15 in areas of transition to saturation. These stripes stem 
from the different color spectra of the a, 0, y, and 6 
pixels, as a result of which the cr+y value of Y(i(even) , j) 
may, for instance, reach saturation at a lower level of 
optical intensity than the 0+6 value of the vertically 

20 adjacent Y(i+l(odd) , j) . Y vlp averages these values to obtain 
a function that is smooth over the transition area. 

Generate output intensity block 48 receives intensity 
(Y) information from intensity evaluation block 44 and 
vertical low-pass intensity (Y vlp ) information from vertical 

25 low-pass filter 52. The output of block 48 is output 
intensity (Y^) to point processing LUT block 62 (see Figure 
1) - 

Block 48 replaces the original luminance Y, computed 
by the intensity evaluation block 44 f with Y vlp when Y vlp 
30 approaches saturation, in order to prevent the appearance 
of horizontal stripes as explained above. Block 48 
implements the function: 
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= ( Y if < ^tfareihoU 

I Y vlp if Y vlp > Yfc.fc.M 

The value of is typically equal to approximately 

220 on an 8-bit scale of 0-255. As values of Y approach 
saturation, image detail is lost in any event, so that 
substituting Y vlp in the high range does not adversely affect 
the perceived resolution. Y vlp is used as the selecting 
input in order to ensure a smooth transition. 

Limit block 50 receives intensity (Y) signals from 
intensity evaluation block 44 and generates limited 
luminance Y^. Limit block 50 cuts off the upper range of 
intensity (Y) values that are to be input to edge detection 
block 56, in order to prevent detection of false edges or 
horizontal stripes that can arise in areas of transition to 
saturation. Limit block 50 implements the function: 

Y^ «= min <Y, Y^y 
the value of Y^ is typically equal to approximately 220. 

The output of limit block 50 (i.e., Y w) is delayed by 
delay buffer 54 and received by edge detection block 56 
which outputs edge information for each pixel. 

Edge detector block 56 convolves the value and its 
8 xmmediate neighbors, with a high-pass or edge detecting 
kernel . 

In one embodiment, the 3x3 Laplacian operator may be 



used: 



1 
8 



f-1 -1 -11 
-1 8-1 
l-l -1 -lj 



Alternatively, to accommodate the geometric 
characteristics of the CCD raster and to give greater 
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emphasis to the vertical edges, the following kernel may be 
used: 

D 8 

In an alternative embodiment, the edge detector block 
56 could be implemented as separate horizontal and vertical 
10 convolution operations (such asalx3or3xl matrix) , 
with additional logic to avoid overemphasis of diagonal 
edges. This alternative embodiment is less hardware 
intensive and gives improved picture quality in some 
circumstances. 

15 Edge suppress block 58 receives the vertical low-pass 

intensity (Y vlp ) signals from vertical low-pass filter 52 and 
outputs edge suppression function f^ to edge multiplier 60. 

The edge suppression function varies between 0 and 1 
in the long exposure processing block 24 only. In the 

2 0 short exposure processing block 26, the function is set to 
1, i.e., no edge suppression at this point. The function 
is typically implemented in block 24 in a piecewise linear 
fashion as follows: 



•% -1 ~h 
-2 8 -2 

-% -1 -%J 



25 [ 1 if Y vlp < LOWSAT 

J Y vb - LOWSAT 

f«* c - S 1 - — — if LOWSAT < Y vto < DEEP SAT 

"* c ] DEEPSAT-LOWSAT 9 

[ 0 if DEEPSAT < Y vlp 

30 Typically LOWSAT is set to approximately 190 and 

DEEPSAT to approximately 220. 

Edge multiplier 60 receives input from blocks 56, 58 
and generates suppressed edge ed^ to intensity (Y) result 
calculation. 

35 Edge multiplier 60 multiplies the edge output of the 

edge detector block 56 by the edge suppression function f^ 
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from block 58 to generate an output value ad to Joint 
operations block 64 (see «m,i-= , i 

aultinlin,*- ■ ThS P^O" of this 

multiplrcation is to suppress distorted large edges that 
»ay appear in the Ion, exposure at intensity (V, values 
5 near saturation, at the same time as they aLear in T 
short exposure at lower values of intensity^ . Th 
double appearanoe of sueh edges was found empirically to 
Z'loZl reSUltln9 dlSPlayed *° - -«^"sLed 

oXTh. shTA 1,9 eicposure 6396 ls suppresee ° S ° *>" 

outL ^osure edge will pass through to the 

output image. The edge suppress function say also be used 
whichlT ™'J*>"™° - — s from the long expose 
15 of « , otherwise exaggerated due to the higher gain 

o„ t , Adaitlonally ' as sl >°™ i" Phantom i„ Figure 5 „ 
optional multiplier or LUT (block s-71 _ ._ 

multiply the output of block 5 times the »tio of "exT *° 
times Cduration of long exposure^":: 

of the exposure and/or gain ratio. This reflects 

of scales of these two values. r "««ots the ratio 

1^^ ^dae'T' * " ath " ° UtPUtS 

25 a Z ZZl ' eQge - *° POlnt Passing block 62 

25 end joint operations block 64 

functions, all of which ^ve ^ut'Ttens^TT 

4 26 (see Figure i) . These functions may typicallv b* 
Elemented as LUTs in RAM or ROM memory. Point^roc* 

6 2 generates arbitrary Action IJ^ZTll 
the joxnt operations block 64 (Figure 1) . 
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The four tables of block 62 are: 

1. The intensity (DC result) block 66 which 

generates a LOT value of intensity (Y^) for the joint 
operations block 64. 
5 Block 66 controls the amount of point ("DC") 

luminance that is summed with the edge information in 
generating the output luminance , Y^. In its most 
general formulation, 

10 Y ta - f (Y^ # Y^ ) 

where f is an arbitrary function. It has been found 
that a quasi logarithmic or fractional power dependence 
of Y^ on the inputs gives the best output image 
appearance, and the general function above can 
15 generally be reduced to a more compact LOT or 

piecewise linear implementation. 

One simple possible computation of Y^ is as 

follows: 

a) Y^ is multiplied by the exposure ratio, so that 
20 it is on the same scale as Y^. That is, if a 

certain pixel x is acquired within the active 
(linear) sensitivity region of both the short and 
long exposures, then Y^x) = I^Y^fx) , where R 
is the exposure ratio, R=long exposure time/ short 
25 exposure time (or any other ratio representing 

the two sensitivities) . 

b) Subsequently, Y^ and ^Y,^ are linearly 
combined, so that the sum of their relative 
weights is always 1. That is, Y wdr =a*Y loog + 

3 0 b*R*Y ahort , l£a>0, l£b>0, and a+b=l (the wdr index 

stands for 'wide dynamic range'). The common 
practice is to set a=l in the region where the 
short exposure is cut-off (too dark) , b=l in the 
region where the long exposure is saturated (too 
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bright), and a>0, b>0 in the region where both 
exposures carry meaningful information. However, 
this does not cover all cases, e.g. when neither 
exposure carries any information (long is 
5 saturated and short is cut-off, or both 

saturated, or both cut-off) . 
c) Finally, the dynamic range of is reduced 

(yielding Y^) by either a logarithmic function, 
or by multiplying it by a small fraction, or by 
10 using any empirically found mapping which 

resembles the log function or a similar contrac- 
tion, e.g., the square root. 
Other possible values for Y te comprise empirical 
modifications of the function described above. 
15 2- & 3. Color weight normalize blocks 68, 70 for 

long and short exposures, respectively, which 
generate normalizing color weights w,/Y, and w,/Y,. 
Blocks 68 and 70 control the proportions' of 
mixing the color values, dr and db, from the long 
and short exposures, respectively, that will be 
used to generate the output color values, dr,^ 
and db^. Generally, w, and w, are chosen so as 
to give predominant weight at each pixel to the 
color values taken from the exposure in which the 
intensity (Y) luminance values are in the linear 
portion of the range, and to give a smooth 
transition over luminance gradient regions of the 
image. For the most part, w, and w, are deter- 
mined on the basis of Y^dong) alone, except for 
cases where the long exposure is near saturation 
while the short is near cutoff, so that neither 
gives a linear reading. 

The weighting values are complementary, 
i.e., w, = l - w,, and w„ w, > o. The outputs of 
blocks 68 and 70 are normalized by division by 
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the corresponding values of for the long and 
short exposures. Preferably , a floating point 
representation for the output values of blocks 
68 , 70 is used so as to maintain sufficient 
accuracy to prevent noticeable quantization in 
the output image. 

Alternatively , instead of weighted addition 
of the normalized color from the two exposures, 
simple selection of the normalized color from one 
exposure or the other, with an ordered dither 
(alternation) of the color selection in areas of 
transition may be used. In this case, when w, = 
1, the color is selected from the long exposure; 
when w g = 1, it is selected from the long 
exposure, and when neither w factor is 1, color 
values are taken alternately from long and short, 
according to a pseudo-random probability 
distribution in which the long and short color 
value probabilities are equal to the Wi and w a 
values. A normalized color value that is an 
average of the long and short values may also be 
mixed into the dither, in order to give a 
smoother color transition* 

Saturation color suppression factor block 72 
generates the color suppression factor Wht that 
reduces chroma saturation (adds white to the 
image) in areas of luminance saturation of the 
input image. An additional edge color 

suppression factor, Z^, is computed in the joint 
operations block (as will be described herein- 
after) . The minimum of Wht and Z u , both of which 
vary from 1 to 0, multiplies the chroma compo- 
nents at the output stage of color conversion. 
Thus, as Wht approaches zero, so does the color 
saturation of the output image. 
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The purpose of the saturation color 
suppression function is to reduce the appearance 
of color artifacts that arise due to CCD 
saturation. The linear relationships between the 
a, 0, 7, and S CCD outputs and the true RGB 
colors break down as the CCD 14 approaches 
saturation. As non-linear deviations cannot be 
readily corrected, suspected distorted colors are 
"whitewashed". Similar techniques are used in 
the analog domain in conventional CCD cameras. 
As shown in Figure 6 : 

Wht - w, + w,*z, 

w, and w, are identical to the above color 
weighting values. The variable 2. is a function 
15 ot Y^C short), varying between 0 and l, as shown 

schematically in the lower right corner of Figure 
6. it tends to zero in areas where the short 
exposure luminance approaches either saturation 
or cutoff. This function will give Wht = o at 
the saturation end (where generally w, = i while 
w, ■ - 0) . At the cutoff end, normally w, * i as 
long as there is adequate overlap between the 
long and short exposures, so that in. this range 
the function will usually give Wht » i. 

In normal mode, in which only one input 
channel is operative (see explanation below) , Wht 
= 1 from Y=o up to the low saturation threshold 
(typically 190) . From this threshold up to the 
deep saturation limit of Y (typically 220) , wht 
drops linearly to its saturation value of o. in 
replay mode (see below), there is no saturation 
color suppression. 

Figure 7 discloses the joint operations block 64 (also 
see Figure 1) . 
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Joint operations block 64 combines the chrominance and 
luminance data from the long and short exposure processing 
blocks 24, 26 , together with data from point processing 
block 62, to generate a combined Y/dr/db result. Block 64 
5 then converts this result to output in standard RGB or 
Y/Cr/Cb (luminance, chrominance (red) and chrominance 
(blue)) color space. A color suppression factor Z is 
computed and applied to the chrominance outputs in order to 
reduce color artifacts (by reducing chroma saturation) 
10 around edges and areas of luminance signal saturation. 

Joint operations block 64 includes: 

1. The dr # db, Y block 74 (recalling that dr 
and db are the differences between 
successive readings in even and odd lines, 

15 respectively) which receives dr, db values 

from the color path outputs of long and 
short exposure processing blocks 24, 26 
: respectively; ed^ from the intensity (Y) 

path output of long exposure processing 

20 block 24 and edge data from the intensity 

(Y) path output of short exposure processing 
block 26; and Y LUT , w,/Y, and w,/Y, from table 
processing (LUT) block 62 . Block 74 
generates combined intensity Y/dr/db results 

25 to color conversion block 78 (to be 

discussed) . Block 74 will be discussed in 
greater detail hereinafter. 

2. The color suppression factor block 76 which 
receives ed^ and ed,^ from edge detector 

30 block 56 and saturation color suppression 

factor (Wht) from point processing block 62 
and generates chroma suppression factor 2 
for color conversion block 78. Block 76 
will be discussed in greater detail 

3 5 hereinafter . 
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3 . The color conversion block 78 which receives 
^twuit/ dfcmuh from block 74 and z, the 

color suppression factor from block 76 and 
generates R^, g^, and and Cr and cb. 
Block 78 will be discussed in greater detail 
hereinafter. 

The dr, db, Y block 74 is shown in further detail in 
Figure 8. 

Block 74 includes an intensity (Y) calculation which 
is performed by adders 79, 80 and edge limiting block 81. 
Adder 79 receives ed^ (long) data from long exposure 
processing block 24 , and ed^ from short exposure 
processing block 26. These two inputs are added to give 
ed ge«„*, which is then input to the edge limiting block 81. 
Edge limiting is implemented as a piecewise linear function 
with 6 inflection points (A,... A,) and 4 slopes (S,...S 4 ), as 
shown in the upper right inset of Figure 8. Generally the 
inflection points and slopes are chosen so as to enhance 
the smaller edges (i.e. , s 2 and s 3 * l) , while large edges 
20 (edge > A 5 or < Aj ) are suppressed, since these large edges 
come through strongly in the Y LUT contribution anyway, the 
output image has a more pleasing appearance if they are not 
additionally enhanced. A, and A, may be set to o, but it is 
sometimes desirable to set them to small non-zero values in 
25 order to suppress false edges due to noise. The best 
results appear to be obtained with |a,| and |a«| values of 
50 to 60. The best values of the slopes | Si | are typically 
in the range 0.5 to 2, but the hardware allows a greater 
range . 

The edge^j^, output is then summed by adder 80 with the 
Y LOT output of block 62 to obtain the output luminance value 
Additionally, as shown in phantom in Figure 10, adder 
80 may be removed from its location in Figure 8 and placed 



30 
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so thac the output of block 81 is not added to Y,^ until 
just before being added into block 113 A ^, that is, as late 
as possible. 

5 Block 74 further includes a dr, db calculation which 

is performed by the remaining sections of block 74. The 
dr, db calculation receives low-pass color components dr, 
db from the color paths of long and short exposure 
processing blocks 24, 26; w,/Y, and w./Y. from block 62; and 

10 as calculated by adder 80. The dr, db calculation 

outputs dr,^ and db,^. 

The long and short values of dr and db are multiplied 
by the respective normalized color weights, w,/Y, and w./Y. by 
multipliers 82, 84, 86, 88. These normalized, weighted 

15 color values from the two exposures are summed together by 
adders 90, 92 and then multiplied by Y^ by multipliers 94, 
96 to give the scaled values: 

dr^ = Ynauh * jdr, + dr. 
20 r Y, Y, i 

<»W = Y^ * |db, + db. -^-} 

Alternatively, dr^ and db^ may be generated by 
25 selection between the long and short normalized dr and db 
inputs (and possibly their long/ short average values) . 

The color suppression factor block 76 of Figure 7 is 
shown in more detail in Figure 9. 

Maximum value block 100 selects the higher of the two 
30 absolute values of ed^ and ed^ as calculated by absolute 
value blocks 98, 99. The result of the calculation of 
block 100, ed^, is input to edge chroma suppression factor 
block 102 to calculate Z^.. The calculation of Z a is 
implemented as a piecewise linear function, shown in the 
35 upper right corner of Figure 9. As can be seen in Figure 
9, receives a value between Th and 1, given by: 
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if ed attX <El 

+ E2 - El (1 - Th) if El^ed mx <E2 

if E2sed mAX 

Typically, £1=10 and E2=27 have been found to give 
good results. The minimum value of 2ed , Th , is ordinarily 
set to zero, to give complete chroma suppression at very 

store! ed96S * * ° ±S US6d ° nly ±n rSp1 ^ ° f *»«" 

stored in mosaic format (see generate mosaic block 120 

described hereinafter, , i„ „ hich case ^ serves ^ 

color anomalies resulting from the ^interpolation of the 
pixel values. 

lni Thereafter < as shown in Figure 9, minimum value block 
104 selects the minimum of the two color suppression 
factors, 2ed and Wht, thereby determining the edge criterion 
or saturation criterion that should be used to provide the 
required degree of chroma suppression at the given pixel 

Referring now to Figure i 0/ which discloses in detail 
color conversion block 78 of Figure 7, one sees that color 
conversion block 78 receives Y_ dr_ and db_ from block 

RGB and JaZl " ° UtpUtS in both th. 

RGB and Y/Cr/Cb formulations. 



in other words, block 78 takes the interim dynamic 
range enhancement results Y/dr/db, and converts them into 
conventional color components for system output. 

Block 78 includes horizontal low-pass filter 106 which 
receives Y,^ and calculates Y ^r- 
30 block 108. COl ° r matrix 

Horizontal low-pass filter 106 is identical to the 
low-pass color component block 36 in the color path block 
28 (see Figures 3 and 4) . since the dr and db inputs to 
the color matrix 108 have already been low-pass filtered by 
this low-pass filter operator, it is necessary to filter 
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the intensity (Y) value as well in order to prevent color 
artifacts. 

Color matrix block 108 receives (lp) from 

horizontal low-pass filter 106 and dr^ and db^ from block 
74 and generates low-pass RGB color component outputs. 

If one recalls the derivation of Y, dr and db from the 
original a, fi, y and 6 values of the mosaic CCD input: 



10 



15 



a + y ± £ 



dr = 
db = S 



2 

y - a 



- 0 



together with the RGB equivalencies of a, f}, y and S, one 
obtains the following relationships between RGB and the 
Y/dr/db values: 



20 



R 
G 
B 



= 3.5 * 



0.2 0.4 0.1 
0.4 -0.2 0.2 
0.2 -0.1 -0.4 





Y ' 




dr 




db 
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30 



The factor of 3.5 is required for normalization of the 
relation Y = R + 1.5G + B. Due to hardware implementation 
considerations, the color conversion matrix is calculated 
as follows: 



R 
G 
B 



= 0.7 * 



1.0 2.0 0.5 
2.0 -1;0 1.0 
1.0 -0.5 -2.0 





Y ' 




dr 







35 



In this way the matrix multiplication is performed by 
a series of shift/add operations. The multiplicative 
factor 0.7 is combined (by multiplication) with externally 
programmed RGB white balance correction factors as is 
described hereinafter. 
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R3B white balance multipliers 109 A , 109 B , 109 C receive 
low-pass RGB signals from color matrix block 108 and 
generate normalized low-pass RGB signals. 

Multipliers 109 A , 109 B , 109 C multiply each of the RGB 
5 low-pass values by a pre-computed white balance correction 
factor, adjusted by the normalization factor 0.7 required 
by the color matrix calculation. Although conventional RGB 
whxte balancing uses only two multiplicative factors 
correcting R and B while G is held constant, this "short 
10 cut" does not preserve constant Y achromatic luminance. 
This loss of normalization may lead to the appearance of 
artifacts and incorrect luminance in the output. it is 
necessary, therefore, to use three multiplicative factors, 
normalized to preserve constant luminance Y. 
15 The calculation of the correction factors is performed 

off-line by capturing a white image and selectively 
computing average values R, G, B, and Y, excluding pixels 
near saturation or cutoff. From the definition Y = R + 
1.5G + B , it follows that for a corrected pixel i„ the 
0 white image, it should be found that: 



25 



30 
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Prom this relationship one derives the correction factors 
to be used in multipliers 109 A , 109 B , 109 C : 



R factor = — - 1 

3.5 R 

G factor = — i £_ 

3.5 G 

B factor = — 1 1_ 

3.5 B 



Output signal enhancement block no (which includes 
chroma suppression and RGB output functions) receives 
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corrected low-pass RGB color component signals from color 
matrix block 108 via multipliers 109 A , 109 B , 109 c ; from 
block 74; Y,^ (lp) from block 106; and chroma suppression 
factor Z from block 76. 
5 As noted above, the RGB values output from color 

matrix block 108 are low-pass values. High-frequency image 
information is 11 re- injected" into RGB according to the 
following equation (given here only for the R component, 
since the treatment of G and B is identical): 

0 R*, = R* + K*Y fttUh - K*Y<J2> 



K is an arbitrary constant between 0 and 1, chosen 
according to the degree of high-frequency enhancement 
required. Values in the range 0.4 < K < 0.8 are typically 
15 used. 

The addition and subtraction of Y^ values to the RGB 
components can alter the original values of R/G and B/G, 
with the result that the correct hue of the image is not 
preserved. Therefore, in an alternative embodiment, R, G 
20 and B are multiplied by a high-pass enhancement function. 

Since the RGB color component values contain both 
luminance and chrominance information, the chroma 
suppression factor, Z, is best applied to chrominance-only 
components, by adders 113 A , 113 B , 113 c : 



25 


Cr = 


R 


~ Y result 




Cg = 


G 


— ^ result 




Cb = 


B 





Combining these equations with the previous ones for "high 
frequency re-injection", one obtains the following formula, 
3 0 which is implemented as shown in Figure 10 (including 
arithmetic element blocks 114, 115, 116) to obtain Cr from 
Rip • 
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Cr - + K* (Y nguh - yjjg, - Y 



result 



and likewise for Cg and cb. These Cr/Cg/cb values are 
multiplied by Z by multipliers 112 A , H2 B , H2 C . At this 
point, y/cr/Cb output is available directly (using Y = 
Y^) , though it is preferable to add a bias of +12 8 to the 
signed digital outputs cr and Cb in order to convert them 
to positive values for D/A conversion, in the alternative, 
**** can be added back into the chroma-suppressed Cr/Cg/cb 
values (by adders 113 A -113 C ) to obtain the final R.,/^, 
Referring now to Figure 11, which discloses generate 
mosaic block 120 of Figure 1 in more detail, one sees that 
the input of generate mosaic block 120 is T^/G^/B^ from 
color conversion block 78 of joint operations block 64. 
The output of block 120 is the equivalent «, fi, y , S values 
in the format: 



"e, 7cq «eq 7eq 
tt eq 7eq «eq 7«, 

a< 



in order to reduce memory requirements for image 
storage and to allow stored images to be replayed through 
the apparatus 10 for display, the final RGB values from the 
processed image are used to generate equivalent, simulated 
25 mosaic values of a, 0, y , and 6. m this way, only eight 
bxts per pixel of information must be stored, rather than 
the 24 bits of full output information. These mosaic 
values can later be replayed to regenerate the stored 
image. 

30 The simulated mosaic values are generated by the 

following matrix in matrix block 122, based on the color 
equivalencies given hereinabove. 
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fin 
7«, 



0 2 1 
112 
2 11 
12 0 



R 
G 
B 



10 



15 



20 



25 



30 
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The factor of h that multiplies the matrix is used for 
reasons of hardware convenience — in order to ensure that 
oi i fii 7i and * do not overflow the range 0-255 of 8 bits. 
To maintain the normalization relations given hereinabove, 
the factor should actually be 1/3.5. Therefore, in replay 
mode, mosaic white balance block 32 is used to multiply the 
a, fi, y, and 6 values back by 4/3.5 (=8/7) before 
reprocessing. Finally, multiplexer 124 selects which one 
of the four mosaic values to output for each pixel 
according to the table: 



Pixel 





j even 


j odd 


i even 


a 


y 


i odd 


J8 


6 



Apparatus 10 has three modes of operation: normal, 
adaptive sensitivity (AS) , and replay. 

1. Normal node emulates the performance of a mosaic color 
CCD camera without adaptive sensitivity. In this mode only 
the long exposure portion of the pipeline operates. The 
processing functions are limited to decoding the mosaic 
input into conventional color components: Y/Cr/Cb or RGB, 
while additionally performing filtering operations for 
anti-aliasing, detail (edge) enhancement and chroma 
suppression where required. 
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2. Adaptive sensitivity mode uses all the resources of 
the processing pipeline to generate wide dynamic range 
images as described hereinabove. 

3. Replay mode is required for displaying images that 
have been stored in RAM or disk. Apparatus 10 stores these 
images in a regenerated mosaic format in order to save on 
storage memory requirements. Replay mode is similar to 
normal mode, except that most of the enhancement operations 
are not performed: since the stored data have already been 
filtered once, it is for the most part not desirable to 
filter them again. 

The preceding specific embodiments are illustrative of 
the practice of the invention. it is to be understood, 
however, that other expedients known to those skilled in 
15 the art or disclosed herein, may be employed without 
departing from the spirit of the invention or the scope of 
the appended claims. 
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CLAIM S 

What: is Claimed is; 

1. A color wide dynamic range video imaging apparatus 
comprising: 

5 sensor means for providing a plurality of color video 

images of a scene at different exposure levels; 

means for dividing each color video image into 
components; and 

means for processing said components of each of said 
10 plurality of video images to produce a combined color video 
image including image information from said components of 
each of said plurality of color video images by applying 
neighborhood transforms to at least one of said components 
of each of said plurality of video images, 
15 wherein said means for processing includes means for 

calculating point intensity data for said each of said 
plurality of said video images. 

2. The color wide dynamic range video imaging 
20 apparatus of Claim 1, wherein said means for processing 

includes means for calculating color weighting factors. 

3. The color wide dynamic range video imaging 
apparatus of Claim 2, wherein said means for processing 

25 includes means for calculating saturation color suppression 
factors . 

4. The color wide dynamic range video imaging 
apparatus of Claim 1, wherein said means for dividing each 

3 0 color video image into components includes a filter means 
in front of said sensor means and said filter means 
includes filter elements of a plurality of colors, said 
plurality of colors corresponding to said components. 
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5. The color wide dynamic range video imaging 
apparatus of Claim 4, wherein said sensor means includes a 
plurality of pixel sensing elements and 

wherein said filter elements are arranged in a regular 
repeating pattern with each filter element in front of a 
single pixel element of said sensing means. 

6. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein said processing means 
includes means for evaluating an intensity of each color 
component; 

means for evaluating color suppression factors for 
each pixel; and 

means for converting said components into RGB color 
space. 

7. The color wide dynamic range video imaging 
apparatus of Claim 6, wherein said means for evaluating an 
intensity of each color component communicates with a means 
for substituting luminance values when luminance otherwise 
approaches saturation. 

8. The color wide dynamic range video imaging 
apparatus of Claim 6, wherein said means for evaluating an 
intensity of each color component communicates with a means 
for limiting luminance values. 

9. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein said processing means 
includes white balancing means which calculates correction 
factors for said components based upon the intensity of 
said color video images. 

10. The color wide dynamic range video imaging 
apparatus of claim 9, wherein white balancing means 
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calculates an average intensity of said color video images 
excluding saturated pixels and cut-off pixels. 

11. The color wide dynamic range video imaging 
5 apparatus of Claim 9, including vertical low-pass filter 

means receiving intensity and false edge suppression means, 
wherein said false edge suppression means receives 
intensity data from said vertical low-pass filter means and 
calculates false edge suppression factors which are reduced 
10 when said intensity data exceeds a pre-selected saturation 
value. 

12. The color wide dynamic range video imaging 
apparatus of Claim 11/ wherein said false edge suppression 

15 means calculates said false edge suppression factors for an 
image of a longest of said different exposure levels. 

13. The color wide dynamic range video imaging 
apparatus of Claim 12 , further including edge detection 

20 means and wherein an output of said edge detection means is 
multiplied times said false edge suppression factors. 

14. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein color components are 

25 calculated by multiplying an intensity times a sum of 
products of prior color components and output of said means 
for calculating said color weighting factors. 

15. The color wide dynamic range video imaging 
3 0 apparatus of Claim 6, including means for calculating white 

balance correction factors for said components converted 
into RGB color space, wherein said white balance correction 
factors for a given respective RGB component are calculated 
by dividing average overall luminance by a multiple of an 
35 average value of said respective given RGB component. 
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16. The color wide dynamic range video imaging 
apparatus of Claim 15, wherein said average value of said 
respective given RGB component is calculated excluding 
pixels which are substantially near saturation or 

5 substantially near cut-off. 

17. The color wide dynamic range video imaging 
apparatus of Claim 6, including means for converting said 
RGB color space components into mosaic color components 

10 corresponding to colors of said filter elements of said 
filter means. 

18. The color wide dynamic range video imaging 
apparatus of Claim l, wherein said means for processing is 

15 implemented on a single chip. 

19. An imaging apparatus comprising: 

sensor means for providing a plurality of color video 
images of a scene at different exposure levels; 

means for dividing each color video image into 
components; and 

means for processing said components of each of said 
plurality of video images to produce a combined color video 
image including image information from said components of 
each of said plurality of color video images by applying 
neighborhood transforms to at least one of said components 
of each of said plurality of video images, 

wherein said means for processing includes means for 
calculating point intensity data for said each of said 
30 plurality of said video images, and 

wherein said processing means includes white balancing 
means which calculates correction factors for said 
components based upon the intensity of said color video 
images . 
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20- A color wide dynamic range video processing chip 
comprising: 

means for processing components of each of a plurality 
of video images of a scene at different exposure levels to 
produce a combined color video image including image 
information from said components of each of said plurality 
of color video images by applying neighborhood transforms 
to at least one of said components of each of said 
plurality of video images, 

wherein said means for processing includes means for 
calculating point intensity data for said each of said 
plurality of said video images. 
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